Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adds expiration iterator #104

Merged
merged 3 commits into from
Jun 22, 2023
Merged

Conversation

SirTyson
Copy link
Contributor

This adds an ExpirationIterator for managing how core emits expiration meta. We want to limit the amount of entries we expire in a given ledger to not strain downstream systems. This is necessary because there can be an arbitrary number of entries that can have a given expiration ledger, so we have to send meta in batches. The idea of the ExpirationIterator is that we gradually scan the entire BucketList. On each ledger close, we scan maxBytesToScan megabytes of the BucketList and expire up to maxEntriesToExpire entries. These two configuration options ensure that we keep up to date with expired entries while not slowing down close time due to too much disk IO or overburdening downstream systems.

To keep track of where we are in the BucketList while scanning, we write the ExpirationIterator on every ledger close. It's not really a ConfigSettingEntry but that was the most convenient place to put it. We have to store it somewhere on the BucketList so new nodes joining the network know where to start expiration, and it also needs to be updated on each ledger close. I think ConfigSettingEntry is the best way to do that currently even though semantically the iterator is not a config setting.

In previous proposals, we discussed creating a new type of Bucket called the "expiration" Bucket that is produced on merges. We would iterate through specific key bounds on a given expiration bucket based on some modulo arithmetic on the current ledger number. This seemed complicated, and adding a new bucket type was controversial. This approach should be much more straightforward, although we may have to tune the maxEntriesToExpire and maxBytesToScan values over time to not lag behind too much w.r.t. expiring data.

@SirTyson SirTyson requested review from graydon and dmkozh June 13, 2023 01:24
@SirTyson SirTyson force-pushed the temp-entry-expiration branch 2 times, most recently from c46d0cf to 1edc761 Compare June 16, 2023 22:41
@SirTyson SirTyson force-pushed the temp-entry-expiration branch from 1edc761 to e104b2b Compare June 16, 2023 22:44
@dmkozh dmkozh merged commit 2892ca0 into stellar:next Jun 22, 2023
@SirTyson SirTyson deleted the temp-entry-expiration branch June 22, 2023 22:33
void;
} ext;
// max number of entries that emit expiration meta in a single ledger
uint32 maxEntriesToExpire;
Copy link
Contributor

@jayz22 jayz22 Jun 28, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happens if number of expiring entries exceeds this? They still all expire but some of them don't emit meta?
Never mind, your PR description is pretty clear, sorry about that.

sisuresh added a commit that referenced this pull request Sep 8, 2023
* Initial commit

* Create check.yml (#2)

Defines the 'complete' status check.

* populate (#1)

* Incorporate CAP-42 changes (#3)

* Pick up Stellar-internal.x (#4)

* add .gitignore (#5)

* Absorb changes from rs-stellar-xdr repo (#7)

* Flatten contract spec discriminants (#8)

* Namespace the contract spec types (#9)

* Delete Stellar-contract-meta.x (#12)

* CAP-0047 - update names (#6)

* Fix contract spec definitions after rename (#13)

* Add contract meta types (#10)

* Rename BINARY to BYTES (#16)

* Add SCContractCode and update related types (#17)

* Add names to function inputs in contract spec (#18)

* Pick up persistent state changes (#19)

* Fix repeated errorCode in SCStatus (#20)

* Pick up changes incorrectly committed to rs-stellar-xdr (#21)

* Add rest of CAP-0056 (#22)

* Add BytesN as type (#23)

* Add Val as a spec type (#25)

* Add name of lib to struct and union spec entries (#26)

* Add adverts & demands to StellarMessage (#24)

* Add SST_CONTRACT_ERROR to SCStatusType (#29)

* Remove `ScHash` and `PublicKey` from object type. (#31)

* Add enum as a contract spec type (#34)

* Add error enum as a distinct UDT type from enum (#35)

* Add SCVal success to INVOKE_HOST_FUNCTION_SUCCESS, fix stellar/stellar-protocol#1307 (#36)

* Updated xdr for token wrapper and source auth (#37)

* add ENVELOPE_TYPE_CONTRACT_ID_FROM_ASSET

* add new HostFunction type and HashIDPreimage for using the source account in the host

* add missing enum value (#38)

* rename HOST_FN_CREATE_CONTRACT (#39)

* Add invoker to contract spec (#40)

* Add account id as object type (#41)

* Rename HOST_FN_CREATE_CONTRACT_WITH_SOURCE to HOST_FN_CREATE_CONTRACT_WITH_SOURCE_ACCOUNT (#42)

* Rename ENVELOPE_TYPE_CONTRACT_ID_FROM_SOURCE to ENVELOPE_TYPE_CONTRACT_ID_FROM_SOURCE_ACCOUNT (#43)

* Rename sourceContractID to sourceAccountContractID (#44)

* Add AccountID as type to contract spec (#45)

* move AccountID to avoid circular reference (#46)

* Add new host functions to xdr (#47)

* Add new host functions to xdr

* rename

* Trivial whitespace changes to Stellar-overlay.x to sync with curr (#50)

* XDR side of the contract deployment changes specified in CAP-46-02. (#51)

* XDR side of the contract deployment changes specified in CAP-46-02.

This introduces the ledger entries to store the wasm code and also changes the host functions to support creating/installing the contracts.

* Remove option to create and install contract in the same operation.

This relies on the ability to include multiple Soroban operations in the transaction. We can bring this back if we decide against that.

* Replace BigInt object with {iu}128 (#57)

* Bigint to int128 cont'd -- spec changes (#58)

* Migrate SC_SPEC from BIG_INT to {IU}128

* Make events a list of lists so we can associate events with their operation (#59)

* Make events a list of lists so we can associate events with their operation

* Add a OperationEvents struct

* Update overlay XDR (#62)

* XDR changes for Auth Next. (#65)

- Introduce the `ScAddress` type to generically represent the address in contracts
- Introduce structured authorization data and structured signature payload to use for auth in contracts

* Add fields for docs to the contract spec (#66)

* Make unit/void case explicit and support more tuples in union spec (#67)

* Make unit/void case explicit and support more tuples in union spec

* avoid void

* Fix syntax (#68)

* Add auth errors (#69)

* Add auth errors

* fixup! Add auth errors

* Add nonce to the signature payload (#71)

* Preliminary "remove objects from XDR" change (#70)

* Add diagnosticEvents (#74)

* XDR for upgradeable config entries (#75)

* Generalize configuration-related XDR.

The changes here are summarized in proposed updated to CAP-47 (stellar/stellar-protocol#1291).

- Use unique keys to identify settings
- Only distribute a hash for config upgrades in `StellarValue`
- Add SCP messages for exchanging config upgrade sets

* Remove unused ConfigSetting

---------

Co-authored-by: Dmytro Kozhevin <dmytro@stellar.org>

* Fix U/I128 and U/I256 number representation for consistent sorting (#78)

* Fix 128 and 256 bits number representation

* fixup! Fix 128 and 256 bits number representation

* Add fee and limit-related configuration XDR based on CAP-0046-07. (#79)

Co-authored-by: Graydon Hoare <graydon@pobox.com>

* add config setting for host logic version (#80)

* Meta xdr (#82)

* Add XDR types for contract meta

* generic

* Update Stellar-contract-meta.x

Co-authored-by: Leigh McCulloch <351529+leighmcculloch@users.noreply.github.com>

* Update Stellar-contract-meta.x

* Update Stellar-contract-meta.x

---------

Co-authored-by: Leigh McCulloch <351529+leighmcculloch@users.noreply.github.com>

* Define config settings for contract costs (#81)

* Define cost parameters

* fixup! Define cost parameters

* Move config settings to separate file

* Remove spaces

* Add stellar namespace

* Transaction changes to support Soroban fee model. (#84)

* Transaction changes to support Soroban fee model.

- Moved all the resources (including the footprint) to the transaction level.
- Added host fn batching to compensate for removal of multi-operation tx support
- Did some passing-by naming cleanup (as we never really have time for that)

* Back out the host logic version config setting, will not use it (#85)

* Change cost param type to int64 (#87)

* Add config settings for contract data limits (#90)

* Add `txSOROBAN_RESOURCE_LIMIT_EXCEEDED` to the respective structs. (#91)

* Remove all TransactionResultSetV2 related changes and puts the hash of the events and return values in InvokeHostFunctionResult (#83)

* Remove all TransactionResultSetV2 related changes and puts the hash of TxHashOfMetaHashesSet in the new LedgerHeaderExtensionV2

* Remove hashing of tx meta components

* Remove nested events now that we only have one op per tx

* Return hash of events and return values in InvokeHostFunctionResult

* Err reform (#92)

* Remove comment with improbable suggestion

* Simplify error codes

* update overlay xdr (#94)

* update overlay xdr

* Add comment

* Replace ContractAuth addressWithNonce option with explicit union (#88)

* Replace ContractAuth addressWithNonce option with explicit union (#88)

* Move ContractAuth signatureArgs into Authorization (#89)

* Remove trailing , (#97)

* Changes to auth-related XDR and a bit of cleanup. (#95)

* Changes to auth-related XDR and a bit of cleanup.

- Prepare create contract host fn for using auth next
- Get rid of unnecessary envelopes
- Passing-by cleanup: get rid of `SCVAL_LIMIT`.

* Rename `HostFunctionArgs` back to just `HostFunction`.

* Use `ScAddress` to identify contracts in auth payload.

* Revert invoke host fn return value to be a single value. (#98)

* One more update to return value. (#99)

* State expiration (#93)

* Adds support for state expiration

* Cleanup and refactoring

* Added rent fees and revert footprint changes

* Get rid of floating point math for rent fees

* Add changes for env

* Adapt names to recent consensus on state expiration terminology

* Add state expiration related entries to LedgerCloseMeta as v2

* Move extension point to front of structure.

---------

Co-authored-by: Graydon Hoare <graydon@pobox.com>
Co-authored-by: Siddharth Suresh <siddharth@stellar.org>

* Make `returnValue` optional in meta. (#100)

* Make `returnValue` optional in meta.

This is meaningless for non-Soroban txs.

* Make whole soroban tx meta optional.

Also did some small extension point cleanup.

* Add execution lanes config (#101)

* Add tx limit

* Minor naming tweak

* Small XDR fix. (#102)

* Add costs for new cryptography host functions (#105)

* Switch Soroban nonces from autoincrement to random values with signature expiration. (#103)

* Expiration bump op (#106)

* bump op

* Update thresholds

* Remove mergeable (#108)

* back out ContractCostType::VerifyEcdsaSecp256k1Sig, keeping Recover (#109)

* Add cost types for int256 arithmetics (#110)

* Add footprint to bump expiration name (#111)

* Remove Op postfix for result code (#113)

* Add support for contract instance storage. (#115)

- Store contract instance that includes executable and storage map under a special key
- Make contract executable separate from ScVal

* Add `VmCachedInstantiation` for future use (#116)

* Update XDR comments (#117)

* Remove ContractDataType from ScVal and rename (#118)

* Adds expiration iterator (#104)

* Adds expiration iterator

* Add BucketList size window and remove eviction iterator

* XDR for RestoreFootprintOp (#120)

* XDR for RestoreFootprintOp

* Use extension point instead

* Replace 'restorable' -> 'persistent' for state expiration config to match new terminology (#114)

* Fix comment (#121)

* Add error when accessing expired entry (#122)

* Refactor config XDR to account for dynamic write fees. (#123)

- Remove the flat fee
- Rename the fields that contribute to the write fee for clarity

* Soroban tx set size upgrade support and XDR cleanup. (#124)

- Use dedicated struct for `InvokeContract` host fn
- Replace `ScVec` with `SCVal` or `SCVal<>` depending on the context

* Make SCError into a union to allow user errors to be u32 (#125)

* Remove incorrect comment about Soroban u256 representation (#126)

* Change metadata fee config to only account for contract events. (#127)

Also renamed a couple tx-size related fields for clarity.

* Update `SorobanResources` for consistency with config update. (#128)

* One more metadata->events rename. (#129)

* Adds eviction iterator (#130)

* Remove `ContractCostType::GuardFrame` (#131)

* Clean up and clearify some `ContractCostType`s (#133)

* Add an error for exceeding refundable fee. (#134)

* Add errors for exceeding the refundable fee.

These have to be operation errors, as core is hard-wired to only return `txFAILED` on transaction failure and modifying this behavior is risky/slow.

* Remove `contractEventsSizeBytes` from `SorobanResources`.

This field is almost redundant and only helps to avoid apply-time error in the rare cases when the user sets the events resource, but forgets to set high enough refundable fee. It does, on the other hand, introduce another apply-time failure condition if the contracts emits a bit more events than expected (even if the refundable fee is high enough to cover that).

The total size of the events emitted is still governed by the network setting.

* Remove ScSpecTypeSet and SC_SPEC_TYPE_SET (#136)

* Expiration Entry rework (#137)

* typo (#139)

* expand SCError comments, no functional change (#140)

* expand SCError comments, no functional change

* Update Stellar-contract.x

Co-authored-by: Siddharth Suresh <siddharth@stellar.org>

---------

Co-authored-by: Siddharth Suresh <siddharth@stellar.org>

* rename txSOROBAN_RESOURCE_LIMIT_EXCEEDED to more general txSOROBAN_INVALID (#143)

* Update .github/workflows/check.yml

Co-authored-by: Graydon Hoare <graydon@pobox.com>

---------

Co-authored-by: stellar-terraform <55583252+stellar-terraform@users.noreply.github.com>
Co-authored-by: Graydon Hoare <graydon@pobox.com>
Co-authored-by: Leigh McCulloch <351529+leighmcculloch@users.noreply.github.com>
Co-authored-by: jonjove <31668823+jonjove@users.noreply.github.com>
Co-authored-by: mlo <marta.lohova@gmail.com>
Co-authored-by: Hidenori <hidenori@stellar.org>
Co-authored-by: Jay Geng <junzhe.g@gmail.com>
Co-authored-by: Dmytro Kozhevin <dmytro@stellar.org>
Co-authored-by: Paul Bellamy <paul@paulbellamy.com>
Co-authored-by: mlo <marta@stellar.org>
Co-authored-by: Garand Tyson <garand@stellar.org>
Co-authored-by: Jay Geng <jay@stellar.org>
Co-authored-by: Brian Anderson <andersrb@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants